{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "f5r7tJESsB65"
   },
   "outputs": [],
   "source": [
    "# Copyright 2019 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SwKhClWLsSou"
   },
   "source": [
    "# **Getting Started with AutoML Tables**\n",
    "\n",
    "<table align=\"left\">\n",
    "  <td>\n",
    "    <a href=\"https://colab.sandbox.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/master/tables/automl/notebooks/census_income_prediction/getting_started_notebook.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
    "    </a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/tables/automl/notebooks/census_income_prediction/getting_started_notebook.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
    "      View on GitHub\n",
    "    </a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SATX51N8tFga"
   },
   "source": [
    "## **Overview**\n",
    "[Google’s AutoML](https://cloud.google.com/automl-tables/) provides the ability for software engineers to build high quality models without the need to know how to build, train models, or deploy/serve models on the cloud. Instead, one only needs to know about dataset curation, evaluating results, and the how-to steps."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0h9L9fpts327"
   },
   "source": [
    "![alt text](https://camo.githubusercontent.com/8d5e7fe8fadc1883bf55b4d561d9b68fced463bf/68747470733a2f2f636c6f75642e676f6f676c652e636f6d2f696d616765732f6175746f6d6c2d7461626c65732f6175746f6d6c2d7461626c652e737667)\n",
    "\n",
    "AutoML Tables is a supervised learning service. This means that you train a machine learning model with example data. AutoML Tables uses tabular (structured) data to train a machine learning model to make predictions on new data. One column from your dataset, called the target, is what your model will learn to predict. Some number of the other data columns are inputs (called features) that the model will learn patterns from.\n",
    "\n",
    "In this notebook, we will use the [Google Cloud SDK AutoML Python API](https://cloud.google.com/automl-tables/docs/client-libraries) to create a binary classification model using a real dataset from the [Census Income Dataset](https://archive.ics.uci.edu/ml/datasets/Census+Income).\n",
    "\n",
    "We will provide the training and evaluation dataset, once dataset is created we will use AutoML API to create the model and then perform predictions to predict if a given individual has an income above or below 50k, given information like the person's age, education level, marital-status, occupation etc...\n",
    "\n",
    "For setting up a Google Cloud Platform (GCP) account for using AutoML, please see the online documentation for [Getting Started](https://cloud.google.com/automl-tables/docs/quickstart)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gKoQObu-s1gT"
   },
   "source": [
    "### **Dataset**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ab_lae0MURrk"
   },
   "source": [
    "This tutorial uses the [United States Census Income Dataset](https://archive.ics.uci.edu/ml/datasets/census+income) provided by the [UC Irvine Machine Learning Repository](https://archive.ics.uci.edu/ml/index.php) containing information about people from a 1994 Census database, including age, education, marital status, occupation, and whether they make more than $50,000 a year. The dataset consists of over 30k rows, where each row corresponds to a different person. For a given row, there are 14 features that the model conditions on to predict the income of the person. A few of the features are named above, and the exhaustive list can be found both in the dataset link above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "c4Tj5-KesxSs"
   },
   "source": [
    "### **Costs**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eX3UU0KDP0Un"
   },
   "source": [
    "\n",
    "This tutorial uses billable components of Google Cloud Platform (GCP):\n",
    "\n",
    "* Cloud AI Platform\n",
    "* Cloud Storage\n",
    "* AutoML Tables\n",
    "\n",
    "Learn about [Cloud AI Platform pricing](https://cloud.google.com/ml-engine/docs/pricing),\n",
    "[Cloud Storage pricing](https://cloud.google.com/storage/pricing),\n",
    "[AutoML Tables pricing](https://cloud.google.com/automl-tables/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wLEZGISqBshz"
   },
   "source": [
    "## **Set up your local development environment**\n",
    "\n",
    "**If you are using Colab or AI Platform Notebooks**, your environment already meets\n",
    "all the requirements to run this notebook. If you are using **AI Platform Notebook**, make sure the machine configuration type is **1 vCPU, 3.75 GB RAM** or above. You can skip this step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fowx0guYB-mX"
   },
   "source": [
    "**Otherwise**, make sure your environment meets this notebook's requirements.\n",
    "You need the following:\n",
    "\n",
    "* The Google Cloud SDK\n",
    "* Git\n",
    "* Python 3\n",
    "* virtualenv\n",
    "* Jupyter notebook running in a virtual environment with Python 3\n",
    "\n",
    "The Google Cloud guide to [Setting up a Python development\n",
    "environment](https://cloud.google.com/python/setup) and the [Jupyter\n",
    "installation guide](https://jupyter.org/install) provide detailed instructions\n",
    "for meeting these requirements. The following steps provide a condensed set of\n",
    "instructions:\n",
    "\n",
    "1. [Install and initialize the Cloud SDK.](https://cloud.google.com/sdk/docs/)\n",
    "\n",
    "2. [Install Python 3.](https://cloud.google.com/python/setup#installing_python)\n",
    "\n",
    "3. [Install\n",
    "   virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv)\n",
    "   and create a virtual environment that uses Python 3.\n",
    "\n",
    "4. Activate that environment and run `pip install jupyter` in a shell to install\n",
    "   Jupyter.\n",
    "\n",
    "5. Run `jupyter notebook` in a shell to launch Jupyter.\n",
    "\n",
    "6. Open this notebook in the Jupyter Notebook Dashboard."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hdzNyF4iCdNI"
   },
   "source": [
    "## **Set up your GCP project**\n",
    "\n",
    "**The following steps are required, regardless of your notebook environment.**\n",
    "\n",
    "1. [Select or create a GCP project.](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
    "\n",
    "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n",
    "\n",
    "3. [Enable the AI Platform APIs and Compute Engine APIs.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component)\n",
    "\n",
    "4. [Enable AutoML API.](https://console.cloud.google.com/apis/library/automl.googleapis.com?q=automl)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ac3NIGCMVF9x"
   },
   "source": [
    "### **PIP Install Packages and dependencies**\n",
    "\n",
    "Install addional dependencies not installed in Notebook environment\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3jK7RbsFVHBg"
   },
   "outputs": [],
   "source": [
    "# Use the latest major GA version of the framework.\n",
    "! pip install --upgrade --quiet --user --user google-cloud-automl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kK5JATKPNf3I"
   },
   "source": [
    "**Note:** Try installing using `sudo`, if the above command throw any permission errors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "A37ofoNkR-L7"
   },
   "source": [
    "`Restart` the kernel to allow automl_v1beta1 to be imported for Jupyter Notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vAxYYE3bTr1A"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<script>Jupyter.notebook.kernel.restart()</script>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.core.display import HTML\n",
    "HTML(\"<script>Jupyter.notebook.kernel.restart()</script>\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3Snl2Ja75qMM"
   },
   "source": [
    "## **Set up your GCP Project Id**\n",
    "\n",
    "Enter your `Project Id` in the cell below. Then run the  cell to make sure the\n",
    "Cloud SDK uses the right project for all the commands in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dkz6SRiMCfSX"
   },
   "outputs": [],
   "source": [
    "PROJECT_ID = \"[your-project-id]\" #@param {type:\"string\"}\n",
    "COMPUTE_REGION = \"us-central1\" # Currently only supported region."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dr--iN2kAylZ"
   },
   "source": [
    "## **Authenticate your GCP account**\n",
    "\n",
    "**If you are using AI Platform Notebooks**, your environment is already\n",
    "authenticated. Skip this step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3yyVCJHFSEKG"
   },
   "source": [
    "Otherwise, follow these steps:\n",
    "\n",
    "1. In the GCP Console, go to the [**Create service account key**\n",
    "   page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).\n",
    "\n",
    "2. From the **Service account** drop-down list, select **New service account**.\n",
    "\n",
    "3. In the **Service account name** field, enter a name.\n",
    "\n",
    "4. From the **Role** drop-down list, select\n",
    "   **AutoML > AutoML Admin** and\n",
    "   **Storage > Storage Object Admin**.\n",
    "\n",
    "5. Click *Create*. A JSON file that contains your key downloads to your\n",
    "local environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Yt6PhVG0UdF1"
   },
   "source": [
    "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "q5TeVHKDMOJF"
   },
   "outputs": [],
   "source": [
    "# Upload the downloaded JSON file that contains your key.\n",
    "import sys\n",
    "\n",
    "if 'google.colab' in sys.modules:    \n",
    "  from google.colab import files\n",
    "  keyfile_upload = files.upload()\n",
    "  keyfile = list(keyfile_upload.keys())[0]\n",
    "  %env GOOGLE_APPLICATION_CREDENTIALS $keyfile\n",
    "  ! gcloud auth activate-service-account --key-file $keyfile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "d1bnPeDVMR5Q"
   },
   "source": [
    "***If you are running the notebook locally***, enter the path to your service account key as the `GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "fsVNKXESYoeQ"
   },
   "outputs": [],
   "source": [
    "# If you are running this notebook locally, replace the string below with the\n",
    "# path to your service account key and run this cell to authenticate your GCP\n",
    "# account.\n",
    "\n",
    "%env GOOGLE_APPLICATION_CREDENTIALS /path/to/service/account\n",
    "! gcloud auth activate-service-account --key-file '/path/to/service/account'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zgPO1eR3CYjk"
   },
   "source": [
    "## **Create a Cloud Storage bucket**\n",
    "\n",
    "**The following steps are required, regardless of your notebook environment.**\n",
    "\n",
    "When you submit a training job using the Cloud SDK, you upload a Python package\n",
    "containing your training code to a Cloud Storage bucket. AI Platform runs\n",
    "the code from this package. In this tutorial, AI Platform also saves the\n",
    "trained model that results from your job in the same bucket. You can then\n",
    "create an AI Platform model version based on this output in order to serve\n",
    "online predictions.\n",
    "\n",
    "Set the name of your Cloud Storage bucket below. It must be unique across all\n",
    "Cloud Storage buckets. \n",
    "\n",
    "You may also change the `REGION` variable, which is used for operations\n",
    "throughout the rest of this notebook. Make sure to [choose a region where Cloud\n",
    "AI Platform services are\n",
    "available](https://cloud.google.com/ml-engine/docs/tensorflow/regions). You may\n",
    "not use a Multi-Regional Storage bucket for training with AI Platform."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "both",
    "colab": {},
    "colab_type": "code",
    "id": "MzGDU7TWdts_"
   },
   "outputs": [],
   "source": [
    "BUCKET_NAME = \"[your-bucket-name]\" #@param {type:\"string\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-EcIXiGsCePi"
   },
   "source": [
    "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket. Make sure Storage > Storage Admin role is enabled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NIq7R4HZCfIc"
   },
   "outputs": [],
   "source": [
    "! gsutil mb -p $PROJECT_ID -l $COMPUTE_REGION gs://$BUCKET_NAME"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ucvCsknMCims"
   },
   "source": [
    "Finally, validate access to your Cloud Storage bucket by examining its contents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vhOb7YnwClBb"
   },
   "outputs": [],
   "source": [
    "! gsutil ls -al gs://$BUCKET_NAME"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SDhrFSBHWkgl"
   },
   "source": [
    "## **Import libraries and define constants**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "10-QDqYIWw6w"
   },
   "source": [
    "Import relevant packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hztZMQ-1WlQE"
   },
   "outputs": [],
   "source": [
    "from __future__ import absolute_import\n",
    "from __future__ import division\n",
    "from __future__ import print_function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Uw6hy3ufXFaE"
   },
   "outputs": [],
   "source": [
    "# AutoML library.\n",
    "from google.cloud import automl_v1beta1 as automl\n",
    "import google.cloud.automl_v1beta1.proto.data_types_pb2 as data_types"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "znEditA8uMgi"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from ipywidgets import interact\n",
    "import ipywidgets as widgets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WSSiBpttCCrZ"
   },
   "source": [
    "Populate the following cell with the necessary constants and run it to initialize constants."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "V41T2eEVBCbh"
   },
   "outputs": [],
   "source": [
    "#@title Constants { vertical-output: true }\n",
    "\n",
    "# A name for the AutoML tables Dataset to create.\n",
    "DATASET_DISPLAY_NAME = 'census' #@param {type: 'string'}\n",
    "# The GCS data to import data from (doesn't need to exist).\n",
    "INPUT_CSV_NAME = 'census_income' #@param {type: 'string'}\n",
    "# A name for the AutoML tables model to create.\n",
    "MODEL_DISPLAY_NAME = 'census_income_model' #@param {type: 'string'}\n",
    "\n",
    "assert all([\n",
    "    PROJECT_ID,\n",
    "    COMPUTE_REGION,\n",
    "    DATASET_DISPLAY_NAME,\n",
    "    INPUT_CSV_NAME,\n",
    "    MODEL_DISPLAY_NAME,\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YaErGUWMCA26"
   },
   "source": [
    "Initialize client for AutoML and AutoML Tables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "h34EOO9QC6-D"
   },
   "outputs": [],
   "source": [
    "# Initialize the clients.\n",
    "automl_client = automl.AutoMlClient()\n",
    "tables_client = automl.TablesClient(project=PROJECT_ID, region=COMPUTE_REGION)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "NB4GVL3hbHZV"
   },
   "source": [
    "## **Test the set up**\n",
    "\n",
    "To test whether your project set up and authentication steps were successful, run the following cell to list your datasets in this project.\n",
    "\n",
    "If no dataset has previously imported into AutoML Tables, you shall expect an empty return."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hNh4IfWVbJZl"
   },
   "outputs": [],
   "source": [
    "# List the datasets.\n",
    "list_datasets = tables_client.list_datasets()\n",
    "datasets = { dataset.display_name: dataset.name for dataset in list_datasets }\n",
    "datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "I7nyfefWba32"
   },
   "source": [
    "You can also print the list of your models by running the following cell.\n",
    "​\n",
    "If no model has previously trained using AutoML Tables, you shall expect an empty return."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "aVOPbJaZbc5o"
   },
   "outputs": [],
   "source": [
    "# List the models.\n",
    "list_models = tables_client.list_models()\n",
    "models = { model.display_name: model.name for model in list_models }\n",
    "models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WvYxMMLVdYmd"
   },
   "source": [
    "## **Import training data**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3toFtz7xb5Ws"
   },
   "source": [
    "### **Create dataset**\n",
    "Now we are ready to create a dataset instance (on GCP) using the client method `create_dataset()`. This method has one required parameter, the human readable display name `DATASET_DISPLAY_NAME`.\n",
    "\n",
    "Select a dataset display name and pass your table source information to create a new dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4UYVvFn9b1uQ"
   },
   "outputs": [],
   "source": [
    "# Create dataset.\n",
    "dataset = tables_client.create_dataset(\n",
    "          dataset_display_name=DATASET_DISPLAY_NAME)\n",
    "dataset_name = dataset.name\n",
    "dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3CLctOh7dzcp"
   },
   "source": [
    "### **Import data**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Y7HLiinLd8mE"
   },
   "source": [
    "You can import your data to AutoML Tables from GCS or BigQuery. For this tutorial, you can use the [census_income dataset](https://storage.cloud.google.com/cloud-ml-data/automl-tables/notebooks/census_income.csv) as your training data. We provide code below to copy the data into a bucket you own automatically. You are free to adjust the value of `BUCKET_NAME` as needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ECklv3hAd0F6"
   },
   "outputs": [],
   "source": [
    "GCS_DATASET_URI = 'gs://{}/{}.csv'.format(BUCKET_NAME, INPUT_CSV_NAME)\n",
    "! gsutil ls gs://$BUCKET_NAME || gsutil mb -l $COMPUTE_REGION gs://$BUCKET_NAME\n",
    "! gsutil cp gs://cloud-ml-data-tables/notebooks/census_income.csv $GCS_DATASET_URI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MvMwm0W_hX9I"
   },
   "source": [
    "Import data into the dataset, this process may take a while, depending on your data, once completed, you can verify the status by printing the dataset object. This time pay attention to the example_count field with 32461 records."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cLCqyBHLhagU"
   },
   "outputs": [],
   "source": [
    "# Read the data source from GCS. \n",
    "import_data_response = tables_client.import_data(\n",
    "    dataset=dataset,\n",
    "    gcs_input_uris=GCS_DATASET_URI\n",
    ")\n",
    "print('Dataset import operation: {}'.format(import_data_response.operation))\n",
    "\n",
    "# Synchronous check of operation status. Wait until import is done.\n",
    "print('Dataset import response: {}'.format(import_data_response.result()))\n",
    "\n",
    "# Verify the status by checking the example_count field.\n",
    "dataset = tables_client.get_dataset(dataset_name=dataset_name)\n",
    "dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Hm-nfjv8htja"
   },
   "source": [
    "## **Review the specs**\n",
    "\n",
    "Run the following command to see table specs such as row count."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "nlpqBWuQhwnm"
   },
   "outputs": [],
   "source": [
    "# List table specs.\n",
    "list_table_specs_response = tables_client.list_table_specs(dataset=dataset)\n",
    "table_specs = [s for s in list_table_specs_response]\n",
    "\n",
    "# List column specs.\n",
    "list_column_specs_response = tables_client.list_column_specs(dataset=dataset)\n",
    "column_specs = {s.display_name: s for s in list_column_specs_response}\n",
    "\n",
    "# Print Features and data_type.\n",
    "features = [(key, data_types.TypeCode.Name(value.data_type.type_code)) \n",
    "            for key, value in column_specs.items()]\n",
    "print('Feature list:\\n')\n",
    "for feature in features:\n",
    "    print(feature[0],':', feature[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XeSCFNxiiZqI"
   },
   "outputs": [],
   "source": [
    "# Table schema pie chart.\n",
    "type_counts = {}\n",
    "for column_spec in column_specs.values():\n",
    "  type_name = data_types.TypeCode.Name(column_spec.data_type.type_code)\n",
    "  type_counts[type_name] = type_counts.get(type_name, 0) + 1\n",
    "    \n",
    "plt.pie(x=type_counts.values(), labels=type_counts.keys(), autopct='%1.1f%%')\n",
    "plt.axis('equal')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8M4t4kAjjC4h"
   },
   "source": [
    "## **Update dataset: assign a label column and enable nullable columns**\n",
    "This section is important, as it is where you specify which column (meaning which feature) you will use as your label. This label feature will then be predicted using all other features in the row.\n",
    "\n",
    "AutoML Tables automatically detects your data column type. For example, for the ([census_income](https://storage.cloud.google.com/cloud-ml-data/automl-tables/notebooks/census_income.csv)) it detects `income_bracket` to be categorical (as it is just either over or under 50k) and age to be numerical. Depending on the type of your label column, AutoML Tables chooses to run a classification or regression model. If your label column contains only numerical values, but they represent categories, change your label column type to categorical by updating your schema.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HCHwwp6w0V0g"
   },
   "source": [
    "### **Update a column: Set nullable parameter**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "l_02sFLqkAVN"
   },
   "outputs": [],
   "source": [
    "column_spec_display_name = 'income' #@param {type:'string'}\n",
    "type_code='CATEGORY' #@param {type:'string'}\n",
    "\n",
    "update_column_response = tables_client.update_column_spec(\n",
    "    dataset=dataset,\n",
    "    column_spec_display_name=column_spec_display_name,\n",
    "    type_code=type_code,\n",
    "    nullable=False,\n",
    ")\n",
    "update_column_response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hzwIuNlJkKUI"
   },
   "source": [
    "**Tip:** You can use `'type_code': 'CATEGORY'` in the preceding `update_column_spec_dict` to convert the column data type from `FLOAT64 to CATEGORY`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "T6eM-Vnr0eIf"
   },
   "source": [
    "### **Update dataset: Assign a label**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "oUoisUpSkXbz"
   },
   "outputs": [],
   "source": [
    "column_spec_display_name = 'income' #@param {type:'string'}\n",
    "\n",
    "update_dataset_response = tables_client.set_target_column(\n",
    "    dataset=dataset,\n",
    "    column_spec_display_name=column_spec_display_name,\n",
    ")\n",
    "update_dataset_response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5jC3dgRwfNfA"
   },
   "source": [
    "## **Creating a model**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SRqzumCpmN6l"
   },
   "source": [
    "### **Train a Model**\n",
    "\n",
    "Once we have defined our datasets and features we will create a model.\n",
    "\n",
    "Specify the duration of the training. For example, `'train_budget_milli_node_hours': 1000` runs the training for one hour. You can increase that number up to a maximum of 72 hours `('train_budget_milli_node_hours': 72000)` for the best model performance.\n",
    "\n",
    "Even with a budget of 1 node hour (the minimum possible budget), training a model can take more than the specified node hours\n",
    "\n",
    "If your Colab times out, use `client.list_models()` to check whether your model has been created. Then use model name to continue to the next steps. Run the following command to retrieve your model.\n",
    "\n",
    "    model = tables_client.get_model(model_display_name=MODEL_DISPLAY_NAME)\n",
    "\n",
    "You can also select the objective to optimize your model training by setting optimization_objective. This solution optimizes the model by using default optimization objective. Refer [link](https://cloud.google.com/automl-tables/docs/train#opt-obj) for more details. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ps7B_UuMmW4Y"
   },
   "outputs": [],
   "source": [
    "# The number of hours to train the model.\n",
    "model_train_hours = 1 #@param {type:'integer'}\n",
    "\n",
    "create_model_response = tables_client.create_model(\n",
    "    model_display_name=MODEL_DISPLAY_NAME,\n",
    "    dataset=dataset,\n",
    "    train_budget_milli_node_hours=model_train_hours*1000,,\n",
    "    exclude_column_spec_names=['fnlwgt','income'],\n",
    ")\n",
    "\n",
    "operation_id = create_model_response.operation.name\n",
    "\n",
    "print('Create model operation: {}'.format(create_model_response.operation))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JoBJO8wtIaio"
   },
   "outputs": [],
   "source": [
    "# Wait until model training is done.\n",
    "model = create_model_response.result()\n",
    "model_name = model.name\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pU9iZ3MDnfjw"
   },
   "source": [
    "## **Model deployment**\n",
    "\n",
    "**Important :** Deploy the model, then wait until the model FINISHES deployment.\n",
    "\n",
    "The model takes a while to deploy online. When the deployment code `response = client.deploy_model(model_name=model.name)` finishes, you will be able to see this on the UI. Check the [UI](https://console.cloud.google.com/automl-tables?_ga=2.255483016.-1079099924.1550856636) and navigate to the predict tab of your model, and then to the online prediction portion, to see when it finishes online deployment before running the prediction cell.You should see \"online prediction\" text near the top, click on it, and it will take you to a view of your online prediction interface. You should see \"model deployed\" on the far right of the screen if the model is deployed, or a \"deploying model\" message if it is still deploying. </span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "t3b-fUQMnXI0"
   },
   "outputs": [],
   "source": [
    "tables_client.deploy_model(model=model).result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jDxcqNl8oLuo"
   },
   "source": [
    "Verify if model has been deployed, check deployment_state field, it should show: DEPLOYED"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Ln-ptzgWoMbF"
   },
   "outputs": [],
   "source": [
    "model = tables_client.get_model(model_name=model_name)\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZzVMSahBoVKb"
   },
   "source": [
    "Run the prediction, only after the model finishes deployment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oHBcyEbZoj98"
   },
   "source": [
    "## **Make an Online prediction**\n",
    "\n",
    "You can toggle exactly which values you want for all of the numeric features, and choose from the drop down windows which values you want for the categorical features.\n",
    "\n",
    "Note: If the model has not finished deployment, the prediction will NOT work. The following cells show you how to make an online prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8tnRnMuHoWtl"
   },
   "outputs": [],
   "source": [
    "workclass_ids = ['Private', 'Self-emp-not-inc', 'Self-emp-inc', 'Federal-gov',\n",
    "                 'Local-gov', 'State-gov', 'Without-pay', 'Never-worked']\n",
    "education_ids = ['Bachelors', 'Some-college', '11th', 'HS-grad', 'Prof-school',\n",
    "                 'Assoc-acdm', 'Assoc-voc', '9th', '7th-8th', '12th', 'Masters',\n",
    "                 '1st-4th', '10th', 'Doctorate', '5th-6th', 'Preschool']\n",
    "marital_status_ids = ['Married-civ-spouse', 'Divorced', 'Never-married',\n",
    "                      'Separated', 'Widowed', 'Married-spouse-absent', \n",
    "                      'Married-AF-spouse']\n",
    "occupation_ids = ['Tech-support', 'Craft-repair', 'Other-service', 'Sales', \n",
    "                  'Exec-managerial', 'Prof-specialty', 'Handlers-cleaners', \n",
    "                  'Machine-op-inspct', 'Adm-clerical', 'Farming-fishing', \n",
    "                  'Transport-moving', 'Priv-house-serv', 'Protective-serv', \n",
    "                  'Armed-Forces']\n",
    "relationship_ids = ['Wife', 'Own-child', 'Husband', 'Not-in-family', \n",
    "                    'Other-relative', 'Unmarried']\n",
    "race_ids = ['White', 'Asian-Pac-Islander', 'Amer-Indian-Eskimo', 'Other',\n",
    "            'Black']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0UwPipiHKM5X"
   },
   "outputs": [],
   "source": [
    "sex_ids = ['Female', 'Male']\n",
    "native_country_ids = ['United-States', 'Cambodia', 'England', 'Puerto-Rico', \n",
    "                      'Canada', 'Germany', 'Outlying-US(Guam-USVI-etc)', \n",
    "                      'India', 'Japan', 'Greece', 'South', 'China', 'Cuba', \n",
    "                      'Iran', 'Honduras', 'Philippines', 'Italy', 'Poland', \n",
    "                      'Jamaica', 'Vietnam', 'Mexico', 'Portugal', 'Ireland',\n",
    "                      'France', 'Dominican-Republic', 'Laos', 'Ecuador',\n",
    "                      'Taiwan', 'Haiti', 'Columbia', 'Hungary', 'Guatemala', \n",
    "                      'Nicaragua', 'Scotland', 'Thailand', 'Yugoslavia', \n",
    "                      'El-Salvador', 'Trinadad&Tobago', 'Peru', 'Hong', \n",
    "                      'Holand-Netherlands']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Qe9pjRVfphNR"
   },
   "outputs": [],
   "source": [
    "# Create dropdown for workclass.\n",
    "workclass = widgets.Dropdown(\n",
    "    options=workclass_ids, \n",
    "    value=workclass_ids[0],\n",
    "    description='workclass:'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-PVX23I0ppJ3"
   },
   "outputs": [],
   "source": [
    "# Create dropdown for education.\n",
    "education = widgets.Dropdown(\n",
    "    options=education_ids, \n",
    "    value=education_ids[0],\n",
    "    description='education:', \n",
    "    width='500px'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vf1NZ-cLptHJ"
   },
   "outputs": [],
   "source": [
    "# Create dropdown for marital status.\n",
    "marital_status = widgets.Dropdown(\n",
    "    options=marital_status_ids, \n",
    "    value=marital_status_ids[0],\n",
    "    description='marital status:', \n",
    "    width='500px'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "TTMobEncpxK7"
   },
   "outputs": [],
   "source": [
    "# Create dropdown for occupation.\n",
    "occupation = widgets.Dropdown(\n",
    "    options=occupation_ids, \n",
    "    value=occupation_ids[0],\n",
    "    description='occupation:', \n",
    "    width='500px'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ijUXgjmxp2eb"
   },
   "outputs": [],
   "source": [
    "# Create dropdown for relationship.\n",
    "relationship = widgets.Dropdown(\n",
    "    options=relationship_ids, \n",
    "    value=relationship_ids[0],\n",
    "    description='relationship:', \n",
    "    width='500px'\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "uDg6wIr4p6n3"
   },
   "outputs": [],
   "source": [
    "# Create dropdown for race.\n",
    "race = widgets.Dropdown(\n",
    "    options=race_ids, \n",
    "    value=race_ids[0],                           \n",
    "    description='race:', \n",
    "    width='500px'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1iXcIMSsp-E6"
   },
   "outputs": [],
   "source": [
    "# Create dropdown for sex.\n",
    "sex = widgets.Dropdown(\n",
    "    options=sex_ids, \n",
    "    value=sex_ids[0],\n",
    "    description='sex:', \n",
    "    width='500px'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "v6yQnqh3qC5N"
   },
   "outputs": [],
   "source": [
    "# Create dropdown for native country.\n",
    "native_country = widgets.Dropdown(\n",
    "    options=native_country_ids, \n",
    "    value=native_country_ids[0],\n",
    "    description='native_country:', \n",
    "    width='500px'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wCKiMoVRqF4x"
   },
   "outputs": [],
   "source": [
    "display(workclass)\n",
    "display(education)\n",
    "display(marital_status)\n",
    "display(occupation)\n",
    "display(relationship)\n",
    "display(race)\n",
    "display(sex)\n",
    "display(native_country)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WyVybQzWqQvZ"
   },
   "source": [
    "Adjust the slides on the right to the desired test values for your online prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UXvN6l4bqLUu"
   },
   "outputs": [],
   "source": [
    "#@title Make an online prediction: set the numeric variables{ vertical-output: true }\n",
    "\n",
    "age = 36 #@param {type:'slider', min:1, max:100, step:1}\n",
    "capital_gain = 40000 #@param {type:'slider', min:0, max:100000, step:10000}\n",
    "capital_loss = 559.5 #@param {type:'slider', min:0, max:4000, step:0.1}\n",
    "fnlwgt = 150000 #@param {type:'slider', min:0, max:1000000, step:50000}\n",
    "education_num = 9 #@param {type:'slider', min:1, max:16, step:1}\n",
    "hours_per_week = 40 #@param {type:'slider', min:1, max:100, step:1}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wAuMAc-cqdKZ"
   },
   "source": [
    "Run the following cell, and then choose the desired test values for your online prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GJxIJ4KUqeWV"
   },
   "outputs": [],
   "source": [
    "inputs = {\n",
    "    'age': age,\n",
    "    'workclass': workclass.value,\n",
    "    'fnlwgt': fnlwgt,\n",
    "    'education': education.value,\n",
    "    'education_num': education_num,\n",
    "    'marital_status': marital_status.value,\n",
    "    'occupation': occupation.value,\n",
    "    'relationship': relationship.value,\n",
    "    'race': race.value,\n",
    "    'sex': sex.value,\n",
    "    'capital_gain': capital_gain,\n",
    "    'capital_loss': capital_loss,\n",
    "    'hours_per_week': hours_per_week,\n",
    "    'native_country': native_country.value,\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "rnbCFWCUqsLO"
   },
   "outputs": [],
   "source": [
    "prediction_result = tables_client.predict(model=model, inputs=inputs)\n",
    "prediction_result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XoL8HCRFq9D3"
   },
   "source": [
    "**Get Prediction**\n",
    "\n",
    "We extract the `google.cloud.automl_v1beta1.types.PredictResponse` object `prediction_result` and iterate to create a list of tuples with score and label, then we sort based on highest score and display it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1A8Z5Rf6rGSn"
   },
   "outputs": [],
   "source": [
    "predictions = [(prediction.tables.score, prediction.tables.value.string_value) \n",
    "               for prediction in prediction_result.payload]\n",
    "predictions = sorted(\n",
    "    predictions, key=lambda tup: (tup[0],tup[1]), reverse=True)\n",
    "print('Prediction is: ', predictions[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zn6QGHIcrehh"
   },
   "source": [
    "Undeploy the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "yWLMYtBzrf1S"
   },
   "outputs": [],
   "source": [
    "undeploy_model_response = tables_client.undeploy_model(model=model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pKTxwtiZsL2G"
   },
   "source": [
    "## **Batch prediction**\n",
    "\n",
    "**Initialize prediction**\n",
    "\n",
    "Your data source for batch prediction can be GCS or BigQuery.\n",
    "\n",
    "For this tutorial, you can use:\n",
    "\n",
    "* [census_income_batch_prediction_input.csv](https://storage.cloud.google.com/cloud-ml-data/automl-tables/notebooks/census_income_batch_prediction_input.csv) as input source.  \n",
    "\n",
    "\n",
    "Create a GCS bucket and upload the file into your bucket.\n",
    "\n",
    "Some of the lines in the batch prediction input file are intentionally left missing some values. The AutoML Tables logs the errors in the `errors.csv` file. Also, enter the UI and create the bucket into which you will load your predictions.\n",
    "\n",
    "The bucket's default name here is `automl-tables-pred` to be replaced with your own.\n",
    "\n",
    "**NOTE:** The client library has a bug. If the following cell returns a:\n",
    "\n",
    "`TypeError: Could not convert Any to BatchPredictResult` error, ignore it.\n",
    "\n",
    "The batch prediction output file(s) will be updated to the GCS bucket that you set in the preceding cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mSKRq1XFs-gb"
   },
   "outputs": [],
   "source": [
    "gcs_output_folder_name = 'census_income_predictions' #@param {type: 'string'}\n",
    "\n",
    "SAMPLE_INPUT = 'gs://cloud-ml-data/automl-tables/notebooks/census_income_batch_prediction_input.csv'\n",
    "GCS_BATCH_PREDICT_OUTPUT = 'gs://{}/{}/'.format(BUCKET_NAME,\n",
    "                                                gcs_output_folder_name)\n",
    "\n",
    "! gsutil cp $SAMPLE_INPUT $GCS_BATCH_PREDICT_OUTPUT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bF3NTUzjvrxU"
   },
   "source": [
    "Launch Batch prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MtgmVjuovsoW"
   },
   "outputs": [],
   "source": [
    "batch_predict_response = tables_client.batch_predict(\n",
    "    model=model, \n",
    "    gcs_input_uris=GCS_BATCH_PREDICT_URI,\n",
    "    gcs_output_uri_prefix=GCS_BATCH_PREDICT_OUTPUT,\n",
    ")\n",
    "print('Batch prediction operation: {}'.format(\n",
    "    batch_predict_response.operation))\n",
    "\n",
    "# Wait until batch prediction is done.\n",
    "batch_predict_result = batch_predict_response.result()\n",
    "batch_predict_response.metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pyvmqopCwMD3"
   },
   "source": [
    "## **Cleaning up**\n",
    "\n",
    "To clean up all GCP resources used in this project, you can [delete the GCP\n",
    "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "AiimlaBwwNCt"
   },
   "outputs": [],
   "source": [
    "# Delete model resource.\n",
    "tables_client.delete_model(model_name=model_name)\n",
    "\n",
    "# Delete dataset resource.\n",
    "tables_client.delete_dataset(dataset_name=dataset_name)\n",
    "\n",
    "# Delete Cloud Storage objects that were created.\n",
    "! gsutil -m rm -r gs://$BUCKET_NAME\n",
    "  \n",
    "# If training model is still running, cancel it.\n",
    "automl_client.transport._operations_client.cancel_operation(operation_id)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AzR0uVbY2BmQ"
   },
   "source": [
    "\n",
    "## **Next steps**\n",
    "Please follow latest updates on AutoML [here](https://cloud.google.com/automl/docs/)."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "getting_started_notebook.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
